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WWC Quick Review of the Report “Transforming the High School 
Experience: How New York City’s New Small Schools Are Boosting 
Student Achievement and Graduation Rates ” 1 



What is this study about? 

The study examined whether winning an admissions 
lottery for a small school of choice improved high 
school students’ progress toward graduation and 
graduation rates. 

The study analyzed data on more than 21 ,000 
students in New York City who participated in a 
ninth-grade admissions lottery for a small school 
of choice. Eighth-grade students who participated 
in lotteries during the 2004-05 through 2007-08 
school years were followed through the 2008-09 
school year, resulting in four years of follow-up data 
for the cohort in the 2004-05 lottery, three years for 
the cohort in the 2005-06 lottery, and so on. 

Key measures of students’ progress toward gradua- 
tion were an on-track indicator after the first year of 
high school— which noted whether students earned 
at least 10 credits and had no more than one failing 
semester in a core subject— and total credits earned 
after the second and third years of high school. 
Graduation rates were measured four years after 
students’ scheduled entry into ninth grade. 

The study measured the effect of an offer to enroll in 
a small school of choice by comparing outcomes of 



Features of Small Schools of Choice 



Small, nonselective high schools located in disad- 
vantaged communities that serve approximately 
100 students per grade. 

Emphasis on academic rigor, strong relationships 
between students and teachers, and community 
partnerships. 

Founded in 2002 by planning teams comprised of 
teachers, administrators, and community partners 
that developed the mission and planned curriculum 
for each new school. 

Utilized substantial new resources, such as new 
principals and teachers, partnerships with organiza- 
tions that had experience starting new schools, and 
start-up grants, to support implementation. 



students who won an admissions lottery with those 
of students who lost the same admissions lottery. 2 
Students could enter admissions lotteries for more 
than one school, and approximately one-quarter of 
participating students won one lottery and lost at 
least one other lottery. 



(continued) 



1 Bloom, H. S., Thompson, S. L., & Unterman, R. (201 0). Transforming the high school experience: How New York City’s new small schools are boosting 
student achievement and graduation rates. New York, NY: MDRC. 

2 This quick review reports the results of the intent-to-treat analysis, found in Appendix B of the study report, which measures the effect of the offer to 
attend a small school of choice and is the analysis preferred by the WWC. The main body of the report presents the effect of attending a small school of 
choice. The effects of attending a small school of choice on high school students' progress toward graduation and graduation rates are reported to be 
20-30% larger and are also statistically significant. See the full report for the results of that analysis, which is not rated in this quick review. 

Quick reviews examine evidence published in a study (supplemented, if necessary, by information from author queries) to assess whether that study’s 
design meets WWC evidence standards. Quick reviews rely on the effect sizes and significance levels reported by study authors. 

The WWC rating applies only to the summarized results, and not necessarily to all results presented in the study. 









WWC Quick Review (continued) 



What did the study find? 

The study found that students who won an admis- 
sions lottery for a small school of choice showed 
the following statistically significant improvements in 
graduation prospects, relative to students who lost 
the same admissions lottery: 

After the first year of high school, lottery winners 
were more likely than members of the control group 
to be on track for graduation: 57.9% compared with 
50.5%. 

After the second and third years of high school, 
lottery winners averaged 1 .4 and 1 .3 more credits 
toward graduation than control group members. 

Four years after their scheduled entry into ninth 
grade, 68.1 % of lottery winners graduated com- 
pared with 63.8% of control group members. 

The WWC has reservations about these results 
because carrying out the lotteries using the method 
described in the report may have resulted in nonran- 
dom differences between the study groups. 



WWC Rating 



The research described in this 
report meets WWC evidence 
standards with reservations 3 

Strengths: Three-quarters of students in the analy- 
sis sample were assigned by a random process. For 
the full sample, the assignment process resulted in 
groups that were equivalent on baseline test scores 
and demographic characteristics. 

Cautions: One-quarter of the students were 
assigned by a process that may not have been 
random. Specifically, for students participating in 
multiple lotteries, the probability of winning later 
lotteries (after the first lottery) depended, in part, 
on the probability of winning earlier lotteries. But 
the probability of winning earlier lotteries was not 
random: it was determined by the lotteries to which 
the student chose to apply. 

The authors showed that lottery winners and losers 
in the full analysis sample were equivalent on demo- 
graphic characteristics and test scores, but it is pos- 
sible that other differences between the two groups 
existed that were not accounted for in the analysis. 



3 Although the version of the report reviewed by the WWC did not contain information on the baseline equivalence of the analysis samples, the study 
authors subsequently provided this information to the review team. 
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